FEAT: Capture API response stop reason on MessagePiece metadata - #2340
Merged
varunj-msft merged 4 commits intoAug 12, 2026
Merged
Conversation
Targets already recorded token usage but discarded why generation stopped, and dropped usage entirely on content-filtered responses. Capture the provider's stop reason alongside token usage: finish_reason for Chat Completions, Completions and LiteLLM; status plus incomplete_reason for the Responses API. A base no-op hook on OpenAITarget is called from _handle_content_filter_response, so a filtered response now records the tokens it consumed instead of returning bare metadata. These keys are reserved for the provider. construct_response_from_request merges the request's metadata into every response piece, so all of them are cleared before a capture writes back the subset its own API reports.
Justin Song (jsong468)
approved these changes
Aug 8, 2026
Justin Song (jsong468)
left a comment
Justin Song (jsong468)
left a comment
Contributor
There was a problem hiding this comment.
LGTM! I would just check that metadata is populated correctly when actually sending to these different targets if you haven't already!
Take the reserved keys as explicit keyword parameters instead of an open mapping, so the clearing loop and the writing loop cannot drift apart: an unrecognized key is now a type error at the call site rather than a value that is silently persisted. Add the two requested inline comments in OpenAICompletionTarget explaining why the capture is per piece and why it borrows the Chat Completions usage parser.
The module comment claimed every target clears the reserved keys, which overstates it: a path that never reaches a provider response captures nothing and so clears nothing. Name that boundary where the contract is written down. Also pin the unreserved-key test to the argument it is really about, and note in _capture_response_cost that it has to run after usage capture, which clears the whole token_usage_ prefix.
varunj-msft
force-pushed
the
varunj-msft/10481-Capture-API-Response-Metadata
branch
from
August 12, 2026 02:45
657cff6 to
b39485e
Compare
varunj-msft
deleted the
varunj-msft/10481-Capture-API-Response-Metadata
branch
August 12, 2026 04:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Targets already recorded token usage but discarded why generation stopped, and dropped
usage entirely on content-filtered responses. This captures the provider's stop reason
alongside token usage on
MessagePiece.prompt_metadata:finish_reason— Chat Completions, Completions, LiteLLMstatus+incomplete_reason— Responses APIA base no-op hook
OpenAITarget._capture_response_metadatais called from_handle_content_filter_response, so a filtered response now records the tokens it consumed.These keys are reserved for the provider.
construct_response_from_requestmerges therequest's
prompt_metadatainto every response piece, so a caller-suppliedfinish_reasonwould otherwise be indistinguishable from the real one. All reserved keys are cleared from
every piece before each capture writes back what its own API reported;
token_usage_*iscleared by prefix for the same reason.
set_response_metadatatakes one keyword-onlyparameter per reserved key, so an unrecognized key is a
TypeErrorat the call site ratherthan a value that is silently dropped.
Two bugs fixed along the way:
OpenAICompletionTargetcaptured neither usage norfinish_reason.n>1, each piece now gets its own choice'sfinish_reasonrather than sharing one.Known gap: the HTTP 400 path builds its response without going
through the capture helpers, so a caller-supplied reserved key still survives there. That
path is unchanged by this PR and behaves as it does on
maintoday.Not breaking:
_METADATA_PREFIX→TOKEN_USAGE_METADATA_PREFIXwas private with noexternal callers.
Tests and Documentation
tests/unit: 15,021 passed / 0 failed. Diff coverage 95% (gate is 90%).pre-commit run --all-filesclean, includingty.per-choice
finish_reasonwithn>1, content-filtered responses, "not reported" cases(missing/empty/non-string omitted rather than stored as zeros), and a SQLite round-trip.
finish_reason=stop; a truncated reasoning request →status=incomplete+incomplete_reason=max_output_tokens; and a forged caller-suppliedfinish_reasoncorrectly cleared.